1. Field of Invention
This invention relates to the design of reconfigurable intelligent photonic interconnect, in particular to the design and packaging of an opto-electronic data processing and switching means called a "smart pixel array" which make possible the realization of a reconfigurable intelligent optical interconnect which implements multiple reconfigurable optical communication channels. The multiple reconfigurable optical channels can simultaneously support multiple "one-to-one", "one-to-many", "one-to-all" and "many-to-many" broadcast communication patterns and can be reconfigured dynamically.
2. Discussion of Prior Art
A literature and patent search did not reveal any prior art in optoelectronic data processing and switching means for the reconfigurable intelligent optical backplanes and interconnect as proposed in this application. A discussion of prior arts based on electrical and optical backplanes will be given.
In a computer system a bus is an interconnection topology commonly used to communicate between plug-in modules in the form of a printed circuit board (PCB). The plug-in modules connect to a common communication media (i.e., the bus) on the backplane printed circuit board. The rules that govern the access of the modules to the bus and the data transfers constitute the bus protocol. A typical implementation of a bus includes a set of metal traces embedded in a backplane PCB. Plug-in modules connect to the bus through edge connectors and drive the bus through high power bus transceivers. Modules connected to the bus all reside on the same backplane PCB. Various standards have been developed which define the physical features of the backplane PCB, the mechanical packaging structure and the bus protocols. These bus standards include VME, FutureBus+, Multibus and Nubus standards and typical standards are described in a textbook by J. Di Giacom, "Digital Bus Handbook", McGraw Hill.
There are many practical limitations to the use of the bus topologies. In the backplane PCB implementing a bus, each metal trace electronically interconnects all plug-in modules. A typical backplane PCB based upon a standardized 19 inch wide rack (the commercial VME and FutureBus standards) houses approximately 20 plug-in modules. To transfer data a single plug-in module must act as a bus master. Distributed protocols are used to arbitrate between contending plug-in modules to appoint the bus-master. To transfer data a bus-master must insert the information including address information and data in a series of individual word transfers over the bus. Typically each word contains 32 bits and the duration of the word transfer is determined by the nature of the bus protocol.
When transferring data the master plug-in module must drive the bus traces on the backplane PCB which in turn must drive all the plug-in modules inserted into the backplane PCB, which can be as high as 20 plug-in modules. The capacitive loading on a bus due to the attached modules greatly increases the propagation delay. This directly affects the data transfer rate in most types of data transfer protocols, for example synchronous protocols in which data transfers are centrally clocked, and in compelled asynchronous protocols which require a handshake on every data transfer. The only data transfer which is not heavily impacted by the bus round trip delay caused by capacitive loading is the uncompelled source synchronous bus transfers in which a long burst transfer is clocked by the sender. In such a transfer the data transfer rate is primarily limited by skews between bits and strobe.
The capacitive loading also decreases the impedance of a bus line to a very low value, i.e., approximately 20 ohms. Since a bus driver sees half the bus impedance i.e. 10 ohms, high currents are required to drive the bus at full speed. For example, a 3 volt swing on a bus which is typical for TTL will require 300 milliAmps (3 volts divided by 10 ohms) to drive the bus on the first transition with proper termination. Since most bus drivers are rated at only 50 to 100 MA the bus is typically under terminated and dependent upon multiple reflections to build up the signal to the final level. The reflections take one or more bus round trip delays to settle resulting in a settling time delay that is a significant portion of the transfer cycle time for a bus.
These limitations limit the aggregate bandwidth of the FutureBus+ to approx. 3.2 GByte/sec or 25.6 Gbit/sec, i.e., the sum of all data communications between plug-in modules cannot exceed approx. 25.6 Gbit/sec since this is the maximum capacity of the bus. In addition to low bandwidths, electronic busses suffer from other limitations. Electronic busses lack multiple independent channels and thus they cannot provide the parallelism required by large scale parallel computing and communication systems. Finally, electronic busses are not scalable to interconnect hundreds of plug-in modules since the increasing capacitance, inductance and impedance problems of a larger bus will lower the already low bandwidth of the bus.
These limitations of electrical busses have not gone unnoticed. In U.S. Pat. No. 5,122,691 Balakrishnan describes a bus architecture which reduces the bus round trip delay by incorporating the busses on an integrated circuit called the bus IC. The plug-in modules have point-to-point links connecting them to the bus ICs. These point-to-point links have a relatively small capacitive load so high power bus transceivers are not required. The bus protocol of the system remains unchanged, i.e. a distributed bus protocol such as the FutureBus+ protocol is used to determine a bus master which transfers data over the bus to one or more bus slaves in a series of individual word transfers according to the protocol. However, the bus transfers are now faster due to the decreased capacitive loading, typically by a factor of 2 or 3.
There are number of problems with Balakrishnan's architecture. First, the bus still supports a single channel where all plug-in modules compete for access to the single channel. At most two plug-in modules can communicate over the bus simultaneously, and the individual word transfers over the bus are still sequential--the improvement over the existing bus performance is a factor of 2 or 3. The bus is electrical and as a result it will always suffer from capacitance, inductance and impedance problems. Finally, it is not scalable to interconnect hundreds of plug-in modules.
Current large communication and computing systems have communications requirements in the hundreds of Gigabits/sec or more. The Cray T3D supercomputer communication mechanism is described in "CRAY T3D System Architecture Overview", Sep. 23, 1993, available from Cray Research, Chippewa Falls, Wis. The electronic bus is far too slow to meet the communication demands of a large computing system. In the Cray system processors reside on PCB modules and the modules are interconnected in the form of a 3 dimensional mesh which occupies many large cabinets. Every node has a direct electronic communication link to two nearest neighbors in each of three dimensions. Each communication link is a point-to-point datapath with 16 bits of data and 8 bits of control. The link is clocked at rates of 150 Mbit/per sec., so that the link has a point-to-point data bandwidth of 2.4 Gbit/sec. A large T3D supercomputer may consist of 1,024 nodes arranged in a 8.times.8.times.16 mesh. The bisection bandwidth of a network is defined as the bandwidth which crosses a bisector which cuts the network into two halves of equal size. If we cut the mesh into two equal size smaller meshes of size 8.times.8.times.8, there are 128 communication links joining these two halves. (This figure includes 64 links between the inner sides of the two halves and 64 links which join the outer sides of the two halves). Hence, the bisection bandwidth of the Cray T3D is 128 links times 2.4 Gbit/sec per link or equivalently 307 Gbit/sec.
However, there are many limitations to the electronic interconnects used in current large scale computing and communication networks. These networks have much more bandwidth than a bus by providing multiple independent high bandwidth communication channels. However, the cost of these multiple channels is a large number of electronic wires between cabinets and electronic traces on PCBs. The inductance and capacitance of these wires and traces necessitates the use of high power transceivers which consume large amounts of power. The inductance and capacitance of these channels also limits the maximum clock rate to at best a few hundred Mbit/sec. The electrical channels are also susceptible to electromagnetic interference.
Hamanka describes a passive optical bus in "Optical bus interconnection system using Selfoc lenses", Optics Letters, Vol. 16, No. 16, Aug. 15, 1991. In this passive optical bus the electronic bus transceivers of the plug-in modules of an electronic bus are replaced by optical bus transceivers. The receiving circuit in an electrical backplane is replaced by a photodetector array. The transmitting circuit in an electrical backplane is replaced by a light modulator. The communication medium responsible for the transferring of data is changed from metal traces on a backplane PCB to optical paths through free-space or an optical medium.
However, Hamanaka's passive optical bus still suffers from many disadvantages associated with the electronic bus. The optical bus still supports a single communication channel so that data transfers still occur sequentially over the bus. Hamanaka's passive optical bus does not implement multiple independent optical channels. A data transfer requires the same steps as in an electrical backplane. A bus master is first selected and the bus master then broadcasts data over the optical bus which must be received by all plug-in module PCBs. These PCBs then must perform packet processing to determine whether the packet is addressed to them. This architecture requires that every plug-in module PCB must be able to monitor all data on the optical bus. The limitation that every PCB must monitor all the data on the optical bus will limit the rate at which data can be transmitted over the bus to the rate at which every PCB can receive and process the data.
An integrated circuit (IC) has a limited electronic Input/Output IO bandwidth of typically tens of Gigabits/second due to IC packaging constraints. Each IC package has at most approx. 500 hundred IO pins due to constraints associated with the connections between the IC substrate and the IC package. A discussion of IC packaging limitations is given in L. L. Moresco, "Electronic System Packaging: The Search for Manufacturing the Optimum in a Sea of Constraints", IEEE Transactions on Components, Hybrids, and Manufacturing Technology", Vol. 13, No. 3, September 1990. Two common techniques to interconnect an integrated circuit VLSI die with a package are "wire-bonding" and "tape automated bonding". These techniques are described in R. R. Tummala and E. J. Rymaszewski (Ed.), "MicroElectronics Packaging Handbook", Reinhold, 1989. Each IO pin has a maximum clock rate of typically a few hundred Mbits/second due to capacitance and inductance and crosstalk associated with the connections between the die and the package. The maximum IO bandwidth of a single IC package is the product of the number of pins times the clock rate per pin. The maximum IO bandwidth of a packaged IC is typically in the tens of Gigabits/second.
Hamanaka's architecture requires that optical signals be received on a photodetector array, all converted to electronics, and all routed off the array to an electronic IC for further processing. Hence, the peak bandwidth of Hamanaka's passive optical bus architecture will be limited to the peak IO bandwidth of an electronic integrated circuit, typically tens of Gigabits/second. Hence, Hamanaka's passive optical bus is simply a faster version of a conventional electronic bus and it does not provide the high bandwidth required by large scale computing and communication systems.
Oh and Kostuk have compared an electronic FutureBus+ with an optical FutureBus+ for example using Hamanaka's passive optical backplane in "Comparison of the Performance Characteristics of FutureBus+ with an Optical Backplane", Proceedings of the 1995 International Conference on Optical Computing, March 1995. Their analysis indicate that each individual word transfer over the optical bus will be 3 or 4 times faster than the electrical bus. This conclusion illustrates the problems with passive optical backplanes such as Hamanaka's; the performance improvement over an electronic bus is a mere factor of 3 or 4, which is achievable electronically as well for example by using the bus IC architecture proposed by Balakrishnan in U.S. Pat. No. 5,122,691 and described earlier.
Redmond and Schenfeld have presented a passive reconfigurable optical interconnect in "A Distributed, Reconfigurable Free-Space Optical Interconnection Network for Massively Parallel Processing Architectures" and in a related paper "Experimental Results of a 64 Channel Free-Space Optical Interconnection Network for Massively Parallel Processing", in the Proceedings of the International Conference on Optical Computing, Aug. 22-25, 1994. This passive optical architecture is similar to Hamanaka's optical bus. The receiving circuit in an electrical backplane is replaced by a photodetector array. The transmitting circuit in an electrical backplane is replaced by a Vertical Cavity Surface Emitting (VCSEL) laser array where every laser is individually addressable. The communication medium responsible for the transferring of data is changed from metal traces on a backplane PCB to optical paths through free-space. (Integrated VCSEL arrays are further described by J. Jewell et. al. in "Vertical-Cavity Surface-Emitting Lasers: Design, Growth, Fabrication, Characterization", IEEE Journal of Quantum Electronics, Vol. 27, No. 6, June 1991.)
This architecture suffers from the same limitations as Hamanaka's optical bus and a few new limitations. The authors emphasize that a key principle guiding their architecture is "logic-less" optical operation. They state that optical technology is used only to provide point-to-point high bandwidth connectivity and nothing else. Their architecture does not support broadcast channels, it does not support one-to-many communications over a single channel, it does not support one-to-all communications over a single channel, it does not support simultaneous many-to-many communications over multiple channels. The architecture is slowly reconfigurable as Redmond and Schenfeld argue that reconfiguration is not needed very often. Slow reconfiguration is achieved by supporting a fixed number of configurations in separate electronic switches. Since their architecture is a passive optical interconnect only, there is no notion of addresses or packets inherent to the architecture. The architecture simply implements multiple passive point-to-point interconnects with no broadcasting.
Broadcasting is a fundamental requirement of parallel computing systems which use "snoopy" caches. Snoopy caches are described in J. L. Hennessy and D. A. Patterson, "Computer Architecture: A Quantitative Approach", Beta version, Morgan Kauffman, 1995. Since this architecture cannot support broadcasting it will have no use in computing and communications systems which require efficient broadcasting, such as computing systems using snoopy caches.
Another key limitation of this architecture is the fact that the optical signals which are received on a photodetector array must be converted to electronics and routed off the array package to an external electronic IC package for further processing. This limitation will limit the peak bandwidth of the optical architecture to the IO bandwidth of a single IC which is typically in the tens of Gbits/sec. Another key limitation of this architecture is the fact that every individual laser in the VCSEL laser array is individually accessible from an IO pin on the VSCEL package. This limitation also implies that the peak bandwidth of their passive optical interconnect will be equal to the peak IO bandwidth of an IC package: the peak bandwidth is limited to the bandwidth which can be supplied to one VCSEL array package or removed from one photodetector array package, which is typically in the tens of Gbits/second.
Finally, the architecture will have optical power limitations as the number of receivers increases. The architecture does not allow for the regeneration of optical signals. A fraction of each optical signal is delivered to each photodetector receiver through the use of partially reflective micromirrrors. This technique will allow an optical signal to be delivered to a small number of receivers, but it cannot be used to interconnect a large number of receivers since the original optical signal can only pass through a limited number of partially reflective mirrors before the signal is lost. For example, if each micromirror reduces the optical signal power by only 20%, then after passing through 10 micromirrors the optical signal power is approximately 1 tenth of its original value. Passing through another 10 mirrors will reduce the signal power to 1 one-hundredth of its original value. Hence, the lack of optical signal regeneration will limit scalability. Cloonan describes a photonic switching system based upon smart pixel arrays in U.S. Pat. No. 5,289,303 "Chuted, Optical Packet Distribution Network", Feb. 22, 1994. Cloonan describes the means for distributing packets of data based upon multiple stages of binary switches or nodes which receive data optically, switch data, and transmit data optically, repeatedly through a series of stages of such devices. The switching architecture is "self-routing" which implies an inherent notion of a packet of data with a header which contains addressing information. The nodes perform processing functions which enable them to change their states in order to propagate the connections in the appropriate direction within the switching network. Multiple nodes are implemented on a single smart pixel array using the FET-SEED smart pixel technology which is further described in the patent.
However, the architecture suffers from many disadvantages. The switch does not allow "locality" between communicating modules i.e., all connections must propagate through all stages of the network . The switch has an input side where all data must enter and an exit side where all data must exit. All communications, even those between modules which are physically close together, must enter the switch at the input side, travel through all the stages of devices and exit the switch at the output side, which can require excessive amounts of time. In a large system the length of fiber just to get data to the input side can cause excessive delays and similarly for the output side. Such multistage switch architectures have largely been abandoned by the parallel computing community, where fast communications between neighboring modules is very important. Furthermore, this switch architecture does not support plug-in modules or data transfers between neighboring plug-in modules.
A literature and patent search did not reveal any prior art in reconfigurable intelligent optical backplanes. The searches indicate that the first proposal of an reconfigurable intelligent optical backplane was by the authors of this application in a paper, T. Szymanski and H. S. Hinton, "Architecture of Terabit Free-space Photonic Backplane", in the Proceedings of the International Conference on Optical Computing, Aug. 22-25, 1994 which described aspects of a reconfigurable optical backplane. In addition, a paper by T. Szymanski and H. S. Hinton entitled "A Smart Pixel Design for a Dynamic Free-Space Optical Backplane" in the Proceedings of the IEEE LEOS Summer Topical Meetings, July 1994 described aspects of the smart pixel arrays for such a backplane. Another talk by T. Szymanski and H. S. Hinton entitled "Graph embeddings in a Free-space photonic backplane" was presented at the IEEE International Conference on Applications of Photonics in Technology, Jun. 21-23, 1994. More recently a paper by T. Szymanski, "Intelligent Optical Backplanes", an invited paper presented at the Proceedings of the International Conference on Optical Computing, Mar. 13-16, 1995, described advanced processing functions of intelligent optical backplanes.
Objects and Advantages
Accordingly, several objects and advantages of the proposed smart pixel arrays for an intelligent photonic system are as follows.
(a) The proposed smart pixel arrays can support multiple dynamically reconfigurable high bandwidth broadcast based communication channels between modules. These multiple independent channels allow for many "one-to-one", "one-to-many", "one-to-all" and "many-to-many" broadcast based communications patterns between modules to occur simultaneously. This capability allows the intelligent photonic interconnect system to be much more powerful and flexible than a passive optical point-to-point interconnect which cannot perform broadcasting.
(b) The communication patterns between modules is dynamically programmable under external control and the patterns can be changed in nanoseconds by down loading a control bit stream into the smart pixel arrays. This dynamic programmability allows the dynamically programmable intelligent photonic interconnect to be used in parallel computing systems, where the communication patterns which are needed may rapidly change over time. This capability allows the intelligent photonic interconnect system to be much more powerful and flexible than a passive optical point-to-point interconnect which is very slowly reconfigured (in the order of seconds per reconfiguration).
(c) The smart pixel arrays contain on-chip data processing means which allow the smart pixel array to simultaneously process the hundreds of Gbits/second of optical data as it travels down the optical interconnect. With this capability a smart pixel array can be selective of which data it wishes to extract from the photonic interconnect and forward to the external electronic data processors. This processing capability allows an intelligent optical interconnect system to implement "one-to-many" broadcast communications patterns mentioned earlier. A sending module may specify a list of intended receivers and insert the data onto the intelligent interconnect. The smart pixel arrays will transport and simultaneously process the data and deliver it to all intended receivers and only those intended receivers.
(d) The smart pixel arrays can process the hundreds of Gbits/second of optical data as it travels down the optical interconnect and selectively extract key data from the photonic interconnect for forwarding to the external electronic data processors. This selective reception of data overcomes a major limitation of passive photonic interconnects. In an intelligent photonic interconnect each optical receiver does not need to forward all optical data to an external electrical processor for processing. Hence, the peak bandwidth of an intelligent photonic interconnect is not limited to the IO bandwidth of an IC package. An intelligent optical interconnect can transport terabits of data per second while delivering tens of Gbits/sec to each thereby honoring the packages limited IO bandwidth. Hence, the intelligent photonic interconnect has a much higher peak bandwidth than a passive point-to-point optical interconnect, where every passive photodetector array must route all incoming optical data off chip to an external electrical processor, thereby limiting the peak bandwidth of the passive optical interconnect to the IO bandwidth of an IC package.
(e) Each packaged smart pixel array provides a means of interfacing between the electronic domain where the IO bandwidth is limited to the tens of Gbits/second due to IC packaging constraints, to the high speed optics domain where the peak IO bandwidth is in the hundreds of Gbits/sec and potentially Terabits/sec range. The proposed smart pixel array design therefore allows for the efficient interfacing between a low bandwidth electronic domain and much higher bandwidth optical domain. The proposed smart pixel array design can therefore be used with a variety of existing IC packaging technologies and a variety of smart pixel array technologies.
(f) Due to their dynamic programmability and architecture, the smart pixel arrays can be used to embed all popular point-to-point networks used in parallel computing and communications, including 2 dimensional and 3 dimensional meshes and hypercubes, into the optical interconnect by programming the smart pixel arrays accordingly. Hence the reconfigurable intelligent optical interconnect will be considerably more powerful and flexible than passive optical busses which implement single non-reconfigurable channels.
(g) Due to their dynamic programmability and ability to process data and selectively extract key data, the smart pixel arrays can be used to embed popular switching networks used in parallel computing and communications. Many parallel computers and communication systems rely upon multiple broadcast channels to implement packet switching of data. The proposed smart pixel arrays can be programmed to implement these packet switching schemes based on multiple parallel broadcast channels by embedding multiple parallel broadcast channels into the intelligent optical interconnect. Hence, the reconfigurable intelligent optical interconnect will be considerably more powerful and flexible than passive optical busses which implement single non-reconfigurable channels, and passive optical interconnects which implement only point-to-point interconnects.
(h) Due to their dynamic programmability and flexibility the smart pixels arrays of the same design can be batch fabricated in large quantities while still covering a large market share. Batch fabrication of a large number of such identical devices will lower the per unit cost significantly, since the custom fabrication of individual devices is not cost effective. A single smart pixel design can be used in the field of parallel computing, to embed 2 dimensional and 3 dimensional meshes and hypercubes, and to embed active packet switches based on multiple broadcast channels, by embedding multiple broadcast channels directly into the intelligent optical interconnect. The ability to appeal to a large market share with the same basic design can significantly lower the cost of such devices.
(i) Due to the dynamic programmability of the smart pixels arrays the network embedded into the photonic interconnect can be reconfigured in the presence of faulty PCBs, thereby increasing fault tolerance.
(j) The ability of an intelligent optical interconnect to process hundreds of Gbits/sec and potentially Terabits/sec will improve the performance of large parallel computing systems. Currently, many critical basic functions are implemented in hardware in a large computing system to improve the performance. Such functions include data transfer acknowledgment, synchronization primitives, resource allocation, flow control and error control, and media access control. All of these critical functions can be implemented over channels in the reconfigurable intelligent optical interconnect, by allocating channel bits for these purposes and performing the necessary processing directly in the smart pixel arrays.
(k) The smart pixel arrays regenerate the optical signals that logically pass through them to full optical power levels. Hence, optical signals can logically pass through large numbers of smart pixel arrays without any loss in power. The reconfigurable photonic interconnect made possible through the use of the proposed smart pixel arrays is scalable to hundreds or even thousands of modules, unlike passive optical architectures which do not regenerate signal power levels and are therefore limited to relatively small sizes.
(l) The proposed smart pixel arrays can be optically interconnected using relatively standard optical imaging techniques such as the use of 2 dimensional fiber bundles. Processing modules which are physically separated meters apart can be interconnected with a distributed reconfigurable intelligent optical interconnect using the proposed smart pixel arrays which are themselves optically interconnected using 2 dimensional fiber ribbons.